Optimal benchmark design varies based on costs in financial manipulation.
problem Manipulation of price benchmarks in finance.
method Analyzes empirical pattern and cost structures to determine optimal benchmark design.
result The optimal benchmark depends on the relative sizes of fixed and variable costs.
This work introduces benchmarks for evaluating nanophotonic structures in design simulations.
problem Design and understanding of nanophotonic structures for various applications.
method Development of frameworks and benchmarks for evaluating nanophotonic structures in parametric design problems.
result Strategic use of evaluation fidelity in enhancing structure designs.
Benchmark proposes to assess molecule docking efficiency.
problem Lack of realistic benchmarks for measuring progress in drug design.
method Proposes a docking-based benchmark using SMINA software.
result Graph-based generative models fail to generate high-scoring molecules.
Fidel-TS creates a new benchmark for time series forecasting models.
problem Lack of high-quality benchmarks for time series forecasting models.
method Formalized high-fidelity benchmark principles, including data sourcing integrity, leak-free design, and structural clarity. Created Fidel-TS, a new large-scale benchmark.
result Demonstrated the limitations of prior benchmarks and potential discrepancies in model evaluation.
GeneDisco benchmarks experimental design for drug discovery.
problem Vast experimental design space in drug discovery.
method Machine learning for optimal experimental design.
result Standardised benchmark suite for active learning.
DOCKSTRING simplifies docking simulations for better drug design benchmarks.
problem Lack of meaningful benchmarks for ligand design.
method Open-source Python package for docking scores, extensive dataset, and pharmaceutically-relevant tasks.
result Docking scores are more appropriate benchmarks than simple physicochemical properties.
Bayesian optimization improves molecule design by addressing three pitfalls.
problem Bayesian optimization pitfalls cause poor performance in molecule design.
method Identified and addressed three pitfalls: incorrect prior width, over-smoothing, and inadequate acquisition function maximization.
result Basic BO setup achieves highest performance on PMO benchmark.
QCircuitBench provides a dataset for evaluating AI's ability to design quantum algorithms.
problem Lack of datasets for evaluating AI's capability in designing quantum algorithms.
method Developed a comprehensive benchmark dataset with 120,290 data points, including 25 algorithms and 3 task suites.
result LLMs exhibit consistent error patterns and fine-tuning does not always outperform few-shot learning.
This paper presents a stochastic logic time delay reservoir design. The reservoir is analyzed using a number of metrics, such as kernel quality, generalization rank, performance on simple benchmarks, and is also compared to a deterministic design. A novel re-seeding method is introduced to reduce the adverse effects of…
Recent studies have significantly improved the state-of-the-art on common-sense reasoning (CSR) benchmarks like the Winograd Schema Challenge (WSC) and SWAG. The question we ask in this paper is whether improved performance on these benchmarks represents genuine progress towards common-sense-enabled systems. We make ca…
Automated HPO design using Bayesian optimization and benchmarking.
problem Designing effective hyperparameter optimization algorithms is manual and lacks systematic understanding.
method Formalized space of HPO candidates, Bayesian optimization for search, ablation analysis.
result Simple configurations can perform well in HPO, especially with right parameters.
Benchmark study evaluates 8 clustering methods on 99 UCR time series datasets.
problem Assessing the performance of clustering methods on time series data.
method Examines 8 clustering methods across 3 categories and 3 distance measures on 99 UCR datasets.
result Provides a comprehensive dataset-level assessment of clustering methods.
New benchmarks for RNA 3D structure-function modeling.
problem Lack of standardized benchmarks for RNA deep learning.
method Developed seven benchmark datasets, provided tools for data handling, and offered a user-friendly environment for model comparison.
result Demonstrated utility with baseline results using a relational graph neural network.
RETINA Benchmark evaluates Bayesian deep learning on diabetic retinopathy detection.
problem Reliable uncertainty quantification for deep learning models in medical applications.
method Design and evaluation of a real-world diabetic retinopathy dataset and tasks.
result Benchmarking of Bayesian deep learning methods on diabetic retinopathy detection tasks.
The current Deep Learning (DL) landscape is fast-paced and is rife with non-uniform models, hardware/software (HW/SW) stacks, but lacks a DL benchmarking platform to facilitate evaluation and comparison of DL innovations, be it models, frameworks, libraries, or hardware. Due to the lack of a benchmarking platform, the …
Deep neural networks solve Raven's Progressive Matrices with high accuracy.
problem Testing relational reasoning in machine learning systems.
method Combining Wild Relation Networks with Multi-Layer Relation Networks and introducing Magnitude Encoding.
result Deep neural networks achieve 98.0 percent accuracy, significantly improving over previous methods.
L2O uses machine learning to design optimization methods.
problem Designing efficient optimization methods for specific problem distributions.
method Data-driven approach to automate optimization method design.
result L2O methods are practical for specific problem distributions but fail on out-of-distribution problems.
Paper reviews neuromorphic engineering features and compares analog vs digital systems.
problem Lack of consensus and unclear features in neuromorphic engineering.
method Review of recent work, comparison of machine learning accelerator chips.
result Analog processing and reduced bit precision architectures offer best efficiencies.
Revisits PPO design choices, exposing failure modes and proposing alternatives.
problem Failure modes of standard PPO in new environments.
method Revisits standard PPO design choices, exposes failure modes, and proposes alternative approaches.
result Alternative design choices prevent failure modes in new environments.
MiniHack simplifies creation of complex RL environments.
problem Limited availability of challenging RL benchmarks.
method Develops a sandbox framework for easy RL environment design.
result MiniHack enables rapid creation of diverse RL testbeds.
We introduce Procgen Benchmark, a suite of 16 procedurally generated game-like environments designed to benchmark both sample efficiency and generalization in reinforcement learning. We believe that the community will benefit from increased access to high quality training environments, and we provide detailed experimen…
Enhanced tabular benchmarks for energy-efficient neural architecture search.
problem Energy consumption in deep learning models.
method Introducing EC-NAS, an enhanced tabular benchmark with energy consumption data.
result EC-NAS reveals a balance between energy usage and accuracy in neural architecture search.
Machine learning (ML) needs industry-standard performance benchmarks to support design and competitive evaluation of the many emerging software and hardware solutions for ML. But ML training presents three unique benchmarking challenges absent from other domains: optimizations that improve training throughput can incre…
Model-based reinforcement learning (MBRL) is widely seen as having the potential to be significantly more sample efficient than model-free RL. However, research in model-based RL has not been very standardized. It is fairly common for authors to experiment with self-designed environments, and there are several separate…
New benchmarks for offline RL from diverse datasets.
problem Measuring progress in offline RL due to lack of suitable benchmarks.
method Developed benchmarks tailored for offline RL, focusing on diverse dataset properties.
result Revealed deficiencies in existing offline RL algorithms.
Natural experiment dataset reveals inconsistent treatment effect estimators.
problem Inconsistent results from over 20 estimators on a new dataset.
method Created a benchmark to evaluate estimator accuracy, derived variance formula, introduced new estimator.
result Doubly robust estimators outperform others by orders of magnitude.
This study analyzes NAS benchmarks and finds that only a subset of operations is crucial for generating high-performing architectures.
problem NAS benchmarks lack generability and provide skewed performance distributions, leading to unreliable comparisons.
method Empirical analysis of widely used NAS benchmarks (101, 201, TransNAS-Bench-101) focusing on operation importance and generability.
result Only a subset of operations is necessary to generate architectures close to the upper-bound performance range, and convolution layers have the highest impact.
Research benchmarks LLMs in medical domain to reduce hallucinations.
problem Hallucinations in medical LLMs can lead to incorrect information.
method Developed Med-HALT dataset and testing methods.
result Significant performance differences among LLMs identified.
Drug discovery projects entail cycles of design, synthesis, and testing that yield a series of chemically related small molecules whose properties, such as binding affinity to a given target protein, are progressively tailored to a particular drug discovery goal. The use of deep learning technologies could augment the …
Study characterizes harmful low-fidelity data sources for surrogate models.
problem Identifying which low-fidelity data sources to use in constructing surrogate models.
method Employed benchmark filtering techniques to assess harmful sources using limited data.
result Provided guidelines for using low-fidelity sources in an industrial setting.
JADAI optimizes design and inference for parameter estimation.
problem Parameter estimation with active optimization of design variables.
method Jointly trains a policy, history network, and inference network to minimize posterior error.
result Achieves superior or competitive performance across benchmarks.
Study logical generalization in GNNs using a new benchmark.
problem Understanding how GNNs adapt to new logical tasks.
method Developed GraphLog benchmark suite for logical tasks, evaluated GNNs in supervised, pretraining, and continual learning settings.
result Logical diversity during training affects GNNs' ability to generalize.
Benchmarked over 70 graph clustering algorithms.
problem Lack of comprehensive performance comparison for graph clustering algorithms.
method Evaluated 70+ graph clustering programs for runtime and quality on weighted and unweighted graphs, analyzed ground truth characteristics.
result Supply a start point for engineers and viewpoint for researchers.
CausalGame benchmarks LLM agents' causal thinking in games.
problem Evaluating causal thinking in AI Scientists with LLMs.
method Interactive games with 14 scenarios incorporating selection bias, measurement error, and hidden confounders.
result None of the 30 LLM agents demonstrated reliable causal thinking, with the best model achieving only 68.0% survival.
This paper considers the mean-reverting portfolio design problem arising from statistical arbitrage in the financial markets. We first propose a general problem formulation aimed at finding a portfolio of underlying component assets by optimizing a mean-reversion criterion characterizing the mean-reversion strength, ta…
We introduce SLM Lab, a software framework for reproducible reinforcement learning (RL) research. SLM Lab implements a number of popular RL algorithms, provides synchronous and asynchronous parallel experiment execution, hyperparameter search, and result analysis. RL algorithms in SLM Lab are implemented in a modular w…
This paper considers the mean-reverting portfolio design problem arising from statistical arbitrage in the financial markets. The problem is formulated by optimizing a criterion characterizing the mean-reversion strength of the portfolio and taking into consideration the variance of the portfolio and an investment budg…
Neural Optimal Design of Experiments improves inverse problem solving efficiency.
problem Optimal experimental design in inverse problems.
method Jointly trains a reconstruction model and design variables in a single loop.
result Significantly reduces computational complexity and improves reconstruction accuracy.
We offer an experimental benchmark and empirical study for off-policy policy evaluation (OPE) in reinforcement learning, which is a key problem in many safety critical applications. Given the increasing interest in deploying learning-based methods, there has been a flurry of recent proposals for OPE method, leading to …
Study benchmarks TSC algorithms in distinguishing diffusions using the likelihood ratio test.
problem Benchmarking optimality of TSC algorithms in distinguishing diffusion processes.
method Proposes to benchmark TSC algorithms using the likelihood ratio test (LRT).
result LRT benchmarks are computationally efficient and can be applied to various time series types.
GraphBench creates a unified benchmark for graph learning tasks.
problem Fragmented benchmarking practices and inconsistent evaluation protocols in graph learning.
method Developed a comprehensive benchmark suite with standardized evaluation protocols.
result Established principled baselines for future research in graph learning.
RainfallBench benchmarks GNSS-based precipitation nowcasting models, addressing complex meteorological challenges.
problem Evaluation of precipitation nowcasting models in meteorology is insufficient due to focus on periodic variables.
method RainfallBench dataset and specialized evaluation protocols for multi-scale, multi-resolution, and extreme rainfall events.
result Bi-Focus Precipitation Forecaster (BFPF) enhances rainfall time series forecasting by incorporating domain-specific priors.
Machine Learning (ML) and Deep Learning (DL) innovations are being introduced at such a rapid pace that researchers are hard-pressed to analyze and study them. The complicated procedures for evaluating innovations, along with the lack of standard and efficient ways of specifying and provisioning ML/DL evaluation, is a …
This article provides a thorough meta-analysis of the anomaly detection problem. To accomplish this we first identify approaches to benchmarking anomaly detection algorithms across the literature and produce a large corpus of anomaly detection benchmarks that vary in their construction across several dimensions we deem…
Understanding and reasoning about physics is an important ability of intelligent agents. We develop the PHYRE benchmark for physical reasoning that contains a set of simple classical mechanics puzzles in a 2D physical environment. The benchmark is designed to encourage the development of learning algorithms that are sa…
Language model benchmarks often misrepresent true understanding, revealing vulnerabilities in evaluation methods.
problem Language model benchmarks fail to accurately reflect true language understanding and adaptability.
method Systematic analysis of NLP evaluation frameworks, identifying vulnerabilities in static benchmarks, human evaluation protocols, and LLM-as-judge frameworks.
result Current evaluation methods are unreliable and need improvement to accurately assess LLM performance.
FinMaster benchmarks LLMs in financial tasks, revealing gaps in reasoning.
problem Challenges in financial tasks, including labor-intensive processes and low error tolerance.
method Developed a comprehensive financial benchmark (FinMaster) with three modules: FinSim, FinSuite, and FinEval.
result LLMs struggle with complex financial reasoning, showing significant accuracy drops.
UCFE benchmarks LLMs in financial tasks with human feedback.
problem Evaluating LLMs' financial task performance and user satisfaction.
method Hybrid approach combining human expert evaluations and dynamic interactions.
result Significant alignment between benchmark scores and human preferences (Pearson correlation coefficient of 0.78).