Tiny benchmarks reduce LLM evaluation costs by using fewer examples.
problem Expensive evaluation of LLMs with tens of thousands of examples.
method Developed evaluation tools and tiny versions of popular benchmarks.
result Accurately estimate LLM performance with just 100 curated examples.
Language model benchmarks often misrepresent true understanding, revealing vulnerabilities in evaluation methods.
problem Language model benchmarks fail to accurately reflect true language understanding and adaptability.
method Systematic analysis of NLP evaluation frameworks, identifying vulnerabilities in static benchmarks, human evaluation protocols, and LLM-as-judge frameworks.
result Current evaluation methods are unreliable and need improvement to accurately assess LLM performance.
Fidel-TS creates a new benchmark for time series forecasting models.
problem Lack of high-quality benchmarks for time series forecasting models.
method Formalized high-fidelity benchmark principles, including data sourcing integrity, leak-free design, and structural clarity. Created Fidel-TS, a new large-scale benchmark.
result Demonstrated the limitations of prior benchmarks and potential discrepancies in model evaluation.
New IRT method identifies useful datasets for ML classifier evaluation.
problem Lack of standard evaluation strategy for ML benchmarks.
method Applied Item Response Theory (IRT) to OpenML-CC18 benchmark.
result Not all datasets are useful for evaluating classifiers.
Benchmark study evaluates 8 clustering methods on 99 UCR time series datasets.
problem Assessing the performance of clustering methods on time series data.
method Examines 8 clustering methods across 3 categories and 3 distance measures on 99 UCR datasets.
result Provides a comprehensive dataset-level assessment of clustering methods.
The optimization of algorithm (hyper-)parameters is crucial for achieving peak performance across a wide range of domains, ranging from deep neural networks to solvers for hard combinatorial problems. The resulting algorithm configuration (AC) problem has attracted much attention from the machine learning community. Ho…
This paper investigates task-level evaluation in few-shot learning models.
problem The reliability of evaluating and tuning models trained for individual tasks in few-shot learning is not well addressed.
method The paper measures accuracy of performance estimators, considers model selection strategies, and examines the reasons for evaluator failure.
result Cross-validation with a low number of folds is best for estimating model performance, while large number of folds is better for model selection.
The paper clarifies conditions for using benchmark scores in machine learning.
problem Using benchmark scores to draw scientific inferences about learning problems.
method Developing conditions of construct validity inspired by psychological measurement theory.
result Clarifies conditions under which benchmark scores support diverse scientific claims.
Benchmarking off-policy evaluation methods for complex policies.
problem Lack of comprehensive benchmarks for off-policy evaluation methods.
method Collection of challenging high-dimensional control tasks and datasets.
result Standardized measure of progress for OPE methods.
The current Deep Learning (DL) landscape is fast-paced and is rife with non-uniform models, hardware/software (HW/SW) stacks, but lacks a DL benchmarking platform to facilitate evaluation and comparison of DL innovations, be it models, frameworks, libraries, or hardware. Due to the lack of a benchmarking platform, the …
GraphBench creates a unified benchmark for graph learning tasks.
problem Fragmented benchmarking practices and inconsistent evaluation protocols in graph learning.
method Developed a comprehensive benchmark suite with standardized evaluation protocols.
result Established principled baselines for future research in graph learning.
Temporal information impacts only a fraction of time series datasets, skewing benchmark evaluations.
problem Temporal information's impact on time series classification is often overestimated.
method Permutation tests on UCR archive to identify datasets where temporal info is irrelevant.
result Many tabular datasets perform well without temporal info, skewing benchmark evaluations.
UCFE benchmarks LLMs in financial tasks with human feedback.
problem Evaluating LLMs' financial task performance and user satisfaction.
method Hybrid approach combining human expert evaluations and dynamic interactions.
result Significant alignment between benchmark scores and human preferences (Pearson correlation coefficient of 0.78).
CSTS benchmarks time series clustering by evaluating correlation structures.
problem Lack of validated ground truth for objectively assessing clustering quality.
method Synthetic benchmark CSTS for evaluating correlation structures in multivariate time series data.
result CSTS enables precise diagnosis of methodological limitations in correlation-based time series clustering.
New benchmark protocol evaluates neural network optimizers for efficiency and data shift sensitivity.
problem Benchmarking neural network optimizers with hyperparameter complexity and data shift sensitivity.
method Proposed a new evaluation protocol combining end-to-end and data-addition training efficiency, using bandit hyperparameter tuning and human study validation.
result No clear winner across all tasks, highlighting the complexity of optimizer performance.
Benchmark detects decision-time leakage in financial backtests.
problem Detecting decision-time leakage in financial machine-learning backtests.
method Toggles one evaluation convention at a time around a clean t+1-open reference, holding other factors fixed. result Inflation is highly selective, affecting specific features and execution methods.
HEAR benchmark evaluates audio representations for diverse tasks.
problem Developing a general-purpose audio representation for various tasks.
method Evaluated 29 models across 19 tasks using 16 datasets.
result No single audio representation performs holistically.
Benchmarking recursive collapse claims with a new framework under false-positive control.
problem Evaluating recursive systems for failure patterns and warning claims.
method Developed Loopzero framework for testing recursive failures, specified claim boundaries in Lean, evaluated under FP constraint, and compared with standard detectors.
result No standard detectors or Loopzero's pre-registered quantile detector achieved the required operating point under the false-positive contract.
Study reveals gaps between simulated and real-world treatment effect evaluation metrics.
problem Evaluation of treatment effect estimation models differs between academic and practical settings.
method Comprehensive empirical study comparing semi-simulated benchmarks and real-world datasets.
result Counterfactual metrics do not reliably predict observable metrics, and rankings from simulated benchmarks do not generalize to real-world data.
OceanForecastBench offers a comprehensive benchmark for data-driven ocean forecasting models.
problem Lack of open-source, standardized benchmarks for data-driven ocean forecasting models.
method Proposes OceanForecastBench, a benchmark with high-quality data and evaluation pipeline.
result Offers the most comprehensive benchmarking framework for data-driven ocean forecasting.
This work introduces benchmarks for evaluating nanophotonic structures in design simulations.
problem Design and understanding of nanophotonic structures for various applications.
method Development of frameworks and benchmarks for evaluating nanophotonic structures in parametric design problems.
result Strategic use of evaluation fidelity in enhancing structure designs.
PerturBench benchmarks ML models for cellular perturbation analysis.
problem Standardizing benchmarking in modeling single cell transcriptomic responses to perturbations.
method Modular platform, diverse datasets, metrics, extensive evaluation, rank metrics.
result Simpler models are competitive and scale well with larger datasets.
Randomly guessing weights helps analyze RL benchmarks objectively.
problem Understanding the complexity of reinforcement learning benchmarks.
method Generate policy networks by randomly guessing their parameters, evaluate on benchmarks, and analyze results.
result Small untrained networks can provide a robust baseline for various RL tasks.
Introduces AMLB, an open benchmark for AutoML frameworks.
problem Challenges in comparing AutoML frameworks.
method Open benchmark with 9 AutoML frameworks, 71 classification, 33 regression tasks, multi-faceted analysis, Bradley-Terry trees.
result Differences in AutoML frameworks' performance and trade-offs.
Paper introduces OARF benchmark suite for federated learning systems.
problem Limited diversity in federated learning benchmarks.
method Characterizes OARF benchmark suite with diverse data and applications.
result Federated learning can effectively increase end-to-end throughput.
WRENCH benchmarks weak supervision datasets for machine learning.
problem Lack of standardized evaluation for weak supervision datasets.
method Developed a comprehensive benchmark platform WRENCH.
result Demonstrated the efficacy of WRENCH as a benchmark platform.
Machine learning (ML) needs industry-standard performance benchmarks to support design and competitive evaluation of the many emerging software and hardware solutions for ML. But ML training presents three unique benchmarking challenges absent from other domains: optimizations that improve training throughput can incre…
GIFT-Eval benchmarks time series forecasting models across diverse datasets.
problem Lack of comprehensive benchmarks for evaluating time series foundation models.
method Developed GIFT-Eval, a benchmark with 23 datasets, 177 million data points, and 144,000 time series.
result Promotes evaluation of foundation models across various domains and frequencies.
Study introduces a benchmark suite for evaluating neural MI estimators on real-world unstructured datasets.
problem Lack of comprehensive evaluation methods for neural MI estimators on real-world unstructured datasets.
method Developed a benchmark suite using same-class sampling and a binary symmetric channel trick.
result Showed accurate manipulation of true MI values of real-world datasets.
FinReflectKG - EvalBench benchmarks financial KG extraction from SEC 10-K filings.
problem Lack of universal benchmark and evaluation framework for financial KG construction.
method Agentic and holistic evaluation principles, deterministic commit-then-justify judging protocol, binary and ordinal evaluations.
result Reflection-based extraction outperforms single-pass extraction in comprehensiveness, precision, and relevance.
Evaluation of Bayesian deep learning (BDL) methods is challenging. We often seek to evaluate the methods' robustness and scalability, assessing whether new tools give `better' uncertainty estimates than old ones. These evaluations are paramount for practitioners when choosing BDL tools on-top of which they build their …
New framework assesses graph-learning datasets for better evaluation.
problem Insufficient evaluation of graph-learning datasets and methods.
method Introduces Rings framework for dataset ablations and proposes performance separability and mode complementarity measures.
result Demonstrates utility of Rings framework for graph-learning dataset evaluation.
A framework for evaluating and benchmarking concept drift detection methods
problem Data stream mining challenged by concept drift
method A novel benchmarking framework
result Fair comparisons of drift detection methods
QUACKIE creates a new benchmark for NLP interpretability.
problem Evaluating NLP interpretability methods is challenging due to biased ground truths.
method Formulated a custom classification task from question-answering datasets, generating unbiased ground truths.
result Demonstrated the effectiveness of current interpretability methods on the new benchmark.
Scoping review and benchmarking of synthetic EHR data generation methods.
problem Creating realistic synthetic electronic health records for research and training.
method Conducted a scoping review and benchmarked seven methods on open-source EHR datasets.
result GAN-based methods excel in fidelity and utility, while rule-based methods excel in privacy protection.
INT benchmark tests theorem proving agents' ability to generalize to unseen theorems.
problem Evaluating theorem proving agents' ability to generalize to unseen theorems.
method INT benchmark based on a theorem generation and proof procedure with adjustable knobs for measuring 6 types of generalization.
result MCTS can help agents prove new theorems.
YAHPO Gym introduces a new benchmark for evaluating hyperparameter optimization methods.
problem Evaluating and comparing hyperparameter optimization methods on well-curated benchmark suites.
method Surrogate-based benchmark collection of 14 scenarios, each with multi-fidelity and multi-objective hyperparameter optimization problems.
result Surrogate-based benchmarks produce more faithful results than tabular benchmarks.
Study benchmarks 19 survival models on 34 datasets, finding Cox model still best.
problem Quantitative comparison of survival models on low-dimensional data.
method Comprehensive benchmarking of 19 models on 34 datasets, tuning and evaluating using 6 metrics.
result Cox Proportional Hazards model remains best overall for low-dimensional, right-censored data.
AI benchmarks evaluate football team performance using generative models.
problem Evaluating human performance in complex interactive tasks is error-prone and unreliable.
method Trained Conditional VRNN Model on player and ball tracking data to imitate and predict team interactions.
result Trained model as a useful benchmark for evaluating team performance in football.
DIGEN benchmark provides synthetic datasets for ML algorithm evaluation.
problem Understanding and comparing machine learning algorithms' performance.
method Synthetic datasets generated using 40 mathematical functions to evaluate machine learning algorithms.
result DIGEN resource facilitates understanding why algorithms perform poorly and provides ideas for improvement.
Paper establishes a comprehensive benchmark for ECG time-series analysis.
problem Incomplete understanding of ECG signal properties and limitations in evaluation metrics.
method Categorization of downstream applications, identification of limitations, introduction of a novel metric, benchmarking of time-series models.
result Validation of the effectiveness of the proposed metric and model architecture.
Benchmark evaluates LLM trading agents by masking identifiers to prevent memory leaks.
problem Evaluate LLM trading agents without relying on market memory or noise.
method Data-side masking protocol, Barra-style performance attribution framework.
result LLM agents' returns are largely explained by market and style exposure, not stock selection.
New method improves consistency of reinforcement learning performance evaluations.
problem Inconsistent performance results in reinforcement learning due to flawed evaluation metrics.
method Proposes a new comprehensive evaluation methodology for reinforcement learning algorithms.
result Demonstrates improved reliability of performance measurements for reinforcement learning algorithms.
RETINA Benchmark evaluates Bayesian deep learning on diabetic retinopathy detection.
problem Reliable uncertainty quantification for deep learning models in medical applications.
method Design and evaluation of a real-world diabetic retinopathy dataset and tasks.
result Benchmarking of Bayesian deep learning methods on diabetic retinopathy detection tasks.
TUDataset provides benchmark datasets for graph learning.
problem Lack of meaningful benchmark datasets and standardized evaluation procedures for graph learning.
method Collection and standardization of 120 graph datasets from various applications.
result Standardized evaluation procedures and baseline experiments provided.
A new framework evaluates large language models efficiently and accurately.
problem Evaluation of large language models is challenging due to stochasticity and heterogeneity of benchmarks.
method Interpretable and scalable framework based on Item Response Theory (IRT) and majorization-minimization principle.
result Our method achieves superior scalability and interpretability compared to existing approaches.
This study analyzes NAS benchmarks and finds that only a subset of operations is crucial for generating high-performing architectures.
problem NAS benchmarks lack generability and provide skewed performance distributions, leading to unreliable comparisons.
method Empirical analysis of widely used NAS benchmarks (101, 201, TransNAS-Bench-101) focusing on operation importance and generability.
result Only a subset of operations is necessary to generate architectures close to the upper-bound performance range, and convolution layers have the highest impact.
Proposes CLRS benchmark to evaluate algorithmic reasoning.
problem Difficulty in transferring results across publications due to targeted algorithmic data.
method Develops a comprehensive benchmark covering various algorithmic tasks.
result Demonstrates performance of algorithmic reasoning baselines on the CLRS benchmark.